I'd also, instead of jsut respectfully declining Yoshida, try to make a counter-offer, basically, we'll do all these things, and actively co-operate to boot, but we won't just be left out of the process. Then make talking to Yakahasi and Gasai a part of the conditions of our co-operation.
A solid argument would be that we can't really NOT interact with the village if Yoshida wants us to be actually helpful. At some point they will have to share their customs with us and it's better to start that process earlier than later.
If we can't get the moderates (Gazai, Takahashi) on our side by interacting with them, then this whole enterprise is doomed to fail anyway.
Meeting with two of the moderates face to face, especially with Gazai, is definitely something we can wrangle out of Yoshida. I would not be very surprised if she even expects something like that.
Why send us out here for us to find and use the scroll ourselves and then take it away if we report to him that we found the scroll and a Village of Ninja of no affiliation?
Here's how I see Jiraya's situation:
He has a lot of shit going down and not enough manpower to get things fixed.
He gets some info about a possible summon scroll. It's just basically what ifs and rumors, not really worth checking out, resources for that could be better spend elsewhere.
Some snazzy missing-nin drop on his lap and he's not really sure about their loyalties or what to do with them. He sends them on a test mission and they do OK and don't fuck things up, great.
"I guess I'll just send them down to check out that summon contract rumor. It probably won't lead to anything but at least these guys get a vacation AND they stay out of my hair."
Some time later the missing-nin show up and lo and behold, they found a hidden Hidden Village and a very probable summon contract.
Now, Jiraya could spend resources acquiring the scroll and then give it to these missing-nin (still not sure about loyalties) OR he could get some well performing jounin from Konoha's roster and send them to acquire the scroll through negotiation OR subterfuge.
For Jiraya, getting the summoning contract for a Leaf-nin is way better than giving it to some missing-nin.
Alternatively he might be more interested in the manpower and resources of the village than the summoning scroll, in which case he will probably try to cut a deal with the village and won't give a flying frog whether we get scroll or not.
Only way we get the scroll is if we get it by ourselves or have something huge to use as a lever to get Jiraya to give it to us. Jiraya has zero interest to waste rare and very finite resources on boosting our powerlevels otherwise.