The easiest version of this craft, a simple non-windowed house.
Google image search “paper house luminaries” and find tons of free templates and ideas. Print on plain or patterned paper, cut out windows if you’d like, fold, glue, then add to your tree. Our printer isn’t set up in the apartment, so I sketched up a similar draft of one of the free templates I found then traced with a ruler. Paper trees go from basic to very elaborate. I could probably fill a whole tree of different houses if I had my Exacto blades and a printer for some different templates!

Here are my drafts. Keep on tweaking if you are free handing this project!
If you are printing the template on decorative paper, print on the blank side. If the pattern is on both sides, fold so the template is inward. Also, practice a few houses on regular printer paper before trying on fancy stuff.

Here is the result. I tried to dot fold lines and make cut lines solid… but messed up in a few places.
Pinterest has a ton of paper houses and paper luminaries. Have fun!