Can I NOT invite a Client when connecting Projects to Books?

Can I NOT invite a Client when connecting Projects to Books?

I've setup a Project with a Primary Contact in order to integrate with Books. 

However, my Client got an invite to join the project. I don't want that. I only want to establish the connection so we can invoice with Books.

How do I do that?

Thanks,

Scott