When performing the OS X Server promotion in OS X (Mountain Lion), the Server application creates a default service data directory structure in /Library/Server, while simultaneously preserving any established service data locations that were migrated from the previous installation. This creates the "Custom" service data location, because OS X Server (Mountain Lion) stores all service data on a single volume by default.

For example, you have migrated a server to OS X Server (Mountain Lion) by installing it on a new volume named "Mountain Lion". Your previous Lion Server installation stored service data on a volume called "Data." After completing migration, Server app in OS X Mountain Lion reports the Service Data location is "Custom." When editing that setting, you select the "Data" volume, and receive the "Existing service data is present" dialog.  You do not want to remove the existing data, because it is in use, but you cannot move the default data from the "Mountain Lion" volume to the "Data" volume as long as the existing data is present.