Surprisingly there is little or nothing in the literature on this. I presume this is mainly because in any place where STV is used they are usually sensible enough to use STV for the actual elections!
However, ordering a list by STV is conceptually simple. The most important concept is that it is not a list at all, but a series of contingent groups of elected candidates. If a party wins one seat their elected candidate (top of the list) should be the winner of an election for one place. If a party wins two seats their elected candidates (the top two on the list) should be the winners of an election for two places. And so on. We should try to ensure that, however many are actually elected from the list, they are the candidates who would have won an STV election for that number of places. If there is a choice of methods we should choose the one which most closely approaches this result.
In practice it is sounder to order the list from the bottom up. There are a number of reasons for this. Firstly, the list is finite so the first step is to choose the whole list, by STV. Secondly the special case of AV (for the top place) is the one most prone to anomaly. If the voter knows that the top candidate on the list must be one of the top two, some "tactical" voting considerations are removed.
Take as an example an ordered list of ten people. The top ten people are chosen by STV from all the candidates available, but they are not yet ordered.
To find out how they are ordered, the ballot papers are first recounted (as an election for nine places) to find the top nine, also by STV. The one person who is knocked out of the top nine but was in the top ten is the last on the ordered list. The ballot papers are recounted again to find the top eight. The person knocked out at this stage is placed next to last, ninth. The process continues for smaller and smaller numbers, right down to one. That person is the top of the list, the winner of an alternative vote election.
One reason these principles must be applied is because of the common fallacy that the order of election in an STV count is a significant indicator of political support for the individuals. This is not so. Imagine a case of two candidates with similar views, let's say they are called Blair and Duncan-Smith, and one candidate with quite different views, say called Kennedy. While Kennedy may well be elected first by having more first preference votes, the combined votes of the others could well be higher while each individually has fewer. STV is about candidates each obtaining a quota of votes sufficient for election. Votes are transferred when contingencies occur, like a candidate receiving more than they need (a surplus) or being excluded for lack of support. STV is not about who gets their quota first.
A suggestion that has been made is that the candidates who fail to get on the list (i.e. outside the top ten in this example) should be withdrawn in subsequent counts to determine the order of the ten. However, this is likely to violate the primary principle that, whatever number of candidates from the list that are eventually elected, they must be (as far as possible) the winners of an STV election for that number of places. This is likely not to be the case if most of the candidates are withdrawn before the count. An example gives details of this pitfall.
In practice the procedure of ordering a list is not quite as simple as the principles above. In particular we have to ensure than everyone in the top nine is in the top ten, and so on. This is done by comparing first preference votes (or those at subsequent stages if the same) where two or more candidates drop out at once.
It may be more work for Returning Officers but the system is designed to ensure that the rules distort STV as little as possible. Using computers means that the additional counting is easy to do.
It's no more work for voters, they just order the candidates in their genuine order of preference, as easy as 1-2-3.
Details page, including gender balance procedure