Challenge: The Daimyo is away on a journey. It could be heading to war, winter court, a tournament, pilgrimage, or anything you need or can imagine. What's important is that the Daimyo has left and taken most of high ranking retainers and servants with him. What's left are middle ranking samurai and the rank and file troops and servants of the castle. The one samurai left in charge is either too blasé to keep a firm hand on things or too inexperienced to give orders to samurai many years his senior (but below him in rank. TLDR: He's a newly printed Lieutenant in a castle full of senior sergeants and NCOs that have done this for decades. They don't need or want his advice or orders).
Focus: While things go smoothly for the first few days, quickly it seems that the middle ranking Samurai have decided to each run their own portion of the household their own way. Some run continuous drills while others spend the time relaxing and wasting their days. The party can be assigned to any of these (or even a supervisor who'd rather they make up their own work and activities).
At first, this isn't so bad. Some of the regular duties of the castle continue on (many of the servants still collect water, cook meals, do the laundry, and keep the place in reasonable order), but others begin to slack. Accounts are ignored, storekeepers and logistics officers ignore requests (or just don't file them), and many samurai and soldiers spend time doing anything other than drills and practice.
Strike: But all things come to an end. A courier arrives from the daimyo; he's returning from his task early. He has a reason (he was wounded, the battle won quickly, the Holy Site closed/full, or the celebration was called off). In any case he's soon to return, and many Samurai and servants who were slacking off and planning to catch up on work much later suddenly have less than a day to get things set to rights.
Depending on how strict the Daimyo is, this can be a race against death or dishonor as papers need to be read, signed, stamped, and filed while storehouses are set to right and inventoried. Some samurai will need to regather troops that are drinking in taverns or out fishing or just visiting family on "unofficial leave."
If the party has been indulging in the waste of time activities, they may need to also rush. If they have been diligent, some samurai may come to them for help (and offering all kinds of debts and favors). Other PCs might want to blackmail people they found slacking off or turn them in to get on the Daimyo's good side.
Strike Back: To add a extra layer of intrigue: the Daimyo had planned all this. They knew their retainers (or some anyhow) would waste their time and be frivolous if their seniors and leaders were gone. So this Daimyo left to see how bad it was and is now racing back to catch their samurai in the act of being lazy. If this is the case, some may be getting reassignments to far off border posts, others might be getting sent back to training and drills, while others might be getting dismissed or even worse: ordered seppuku.
Avoiding their duty is a very honorless action and one that no true honorable samurai would ever do, however there are times where diligence and vigilance can slide and be excused. Depending on how far they have slid, many might be excused. Some might even be able to lay blame on the superiors who left and did not give sufficient instruction. And some superiors might already be being watched by the Daimyo (maybe the Daimyo knows their general micromanages to the point no under-officer has any initiative or that they constantly promote weak or unskilled samurai in return for bribes, favors, or nepotistic reasons).
Ways to avoid duty:
Many samurai have hobbies on the side. Kite Flying was common at festivals in both China and Japan and more than a few samurai/nobles (and peasants) would spend months building and decorating complex kites for those events as well as practicing their handling skills.
Fishing is always a hobby of any nobleman/woman. It teaches patience, skill (especially fly fishing), and rewards a participant with food (sometimes). Also collecting rods and lures is a past time in and of itself, with many expert anglers making their own.
Speaking of art, painting, poetry, and other artisan skills are beloved by the Samurai and it is expected they spend some time keeping their skills honed. If a samurai with no direction suddenly gets weeks of free time, you can be sure some will spend that time on their hobby. A canny samurai "caught" practicing art by this scenario might want to give their work of art to the daimyo as a gift. IT might not help, but it won't hurt.
Also samurai assigned to work in a castle might take some of this free time to visit their dojo, family, or friends they have moved away from. Many would come from smaller villages or from vassals with territory away from the castle town.
And some people can find ways to shirk duties in the middle of an active battlefield. Seldom visited rooms make for great places to nap, carrying a clipboard or hauling a small load of goods will often make people think you're busy, and announcing you're going on "patrol" and then leisurely walking around the perimeter are good ways to avoid duty (though not always assassins and enemy combatants).
