Child custody disputes can take place in or out of the divorce context. Regardless of whether or not the parents were ever married, the process and rules involved with a child custody dispute are basically the same.
When a child custody case is not part of a divorce, the initial suit is filed as a Suit Affecting the Parent-Child Relationship or a Suit to Adjudicate Parentage (if paternity has not been legally established). Unlike a divorce, which has a sixty-day waiting period before it can be finalized, there is no waiting period for a pure child custody matter. If the parties have reached an agreement, a final order can be entered almost immediately after filing the suit.
In more amicable cases, the Petitioner (the party who files the suit) will present the Respondent (the person being sued) with the suit and ask him or her to sign a Waiver of Service. The Waiver of Service simply tells the court that the Respondent received the suit and is agreeing not to be formally served. (Most people do not want to be served, so this is the friendlier and cheaper option. I always recommend trying for a waiver first unless the situation is hostile or unless a quick hearing is needed that requires notice.)
Some contested cases need court intervention quickly in the form of a temporary orders hearing. This hearing can often occur within a couple of weeks of filing, and it is sort of a mini-trial. At a temporary orders hearing, the court will determine custody arrangements for while the case is pending and order temporary child support, if appropriate. The court may also order a social study or mediation.
If the case does not settle quickly, discovery will be needed. Discovery can include obtaining records, sending formal written discovery to the other party (Requests for Production of Documents, Requests for Admissions, or Interrogatories), or depositions. In contested custody cases, a social study is often required. Before a final trial, most courts will order the parties to mediate the case. If a settlement is not reached, then the parties will proceed to a final trial.
In my experience, most contested custody disputes last about a year. A settlement can be reached at any time, and the vast majority of cases do settle before a final trial.