If within the alert setup the "Maximum # of documents" is set as 5 and 10 records are retrieved and available for delivery, which 5 records are actually being delivered?

The documents that are delivered for an alerting period, where the number of matching documents exceed the “Maximum # of documents”, is determined by the order in which the documents match. The first documents that match, up to the maximum that is set for the alert, are those that are delivered.

Example for documents loaded to the system during an alerting period:
Loading / matching orderDocument ID
133264876
288929750
360328682
493372879
513995171
697984949
786336645
847042481
963270643
1034155254
 
If the “Max # of documents” is set to 5, documents 1-5 will be delivered in your alert for that alerting period (highlighted in the table). The remaining 5 documents are not delivered.
 
Thus, the order in which the documents are loaded into the search engine and match your alert are the order in which they are delivered, stopping at the “Max # of documents”.