The easiest version of this craft, a simple non-windowed house.
Google image search “paper house luminaries” and find tons of free templates and ideas. Print on plain or patterned paper, cut out windows if you’d like, fold, glue, then add to your tree. Our printer isn’t set up in the apartment, so I sketched up a similar draft of one of the free templates I found then traced with a ruler. Paper trees go from basic to very elaborate. I could probably fill a whole tree of different houses if I had my Exacto blades and a printer for some different templates!
If you are printing the template on decorative paper, print on the blank side. If the pattern is on both sides, fold so the template is inward. Also, practice a few houses on regular printer paper before trying on fancy stuff.
Pinterest has a ton of paper houses and paper luminaries. Have fun!