Hi @jgiessel, welcome to the forum 
Seems like there was an oversight on our part when we updated the install script for the Sub-4.0 tag we forgot to update the ArduSub-stable tag to point to the new Sub-4.0 position.
Until we fix that tag, you should be able to instead do
git checkout Sub-4.0
# create your own branch off of Sub-4.0, called 'new-branch'
git branch new-branch
# move onto your branch
git checkout new-branch
Putting that into the full set of Ubuntu instructions, that’s:
git clone https://github.com/your-github-userid/ardupilot
cd ardupilot
git checkout Sub-4.0
# create your own branch off of Sub-4.0, called 'new-branch'
git branch new-branch
# move onto your branch
git checkout new-branch
# update submodules
git submodule update --init --recursive
# install prerequisites
Tools/environment_install/install-prereqs-ubuntu.sh -y
# reload the path
. ~/.profile
# configure waf for your board (e.g. Pixhawk1)
./waf configure --board Pixhawk1
From there you can make your changes (e.g. create your custom frame), and then compile with
./waf sub