A dev could answer for sure, but given that we specify a query collection interval, I believe that every "x" seconds osqueryd will connect to and poll for any live query jobs. It does same thing with configuration to check for config updates.
The UI doesn't reach out to osquery nodes, the nodes "check in" at the interval you specify in the global options.
Can you send 1 query to 1000s or millions of nodes or is it 1-1 everytime?
Yep, if you have 1000s or millions of nodes checking in to fleet, they will check in and look for queries assigned to them to process.