Next to a large rock are a plaque and a monument to the Genna Christians. It was at this site that Iemitsu Tokugawa, the third Edo shogun, had 50 Christians executed in 1623, and where more Christians were executed 15 years later. It was the setting for Shusaku Endo's short story, "Fuda No Tsuji", centered around the monk "Nezumi" (rat) and a protagonist referred to as "Otoko" (man).

According to the shrine's history, it is one of several ancient shrines in the city used to house the divided spirits (bunrei) of the Ise Shrine during the middle of the Heian Period. It got its name of Shiba-daijingu starting from the Meiji Period, before which it was called by other names including Iikura-shinmeigu and Shiba-shinmeigu. The name of Shiba-shinmeigu is known for the famous "Megumi Fight" that took place in March 1805 inside the shrine, where Kanjin-sumo wrestlers such as Yotsuguruma Daihachi and Mizuhiki Seigorou fought with the Megumi-no-tobi, a group of firefighters who considered the area their territory. Even back then, the Minato ward lying in front of the shrine's gates was a lively business district. During the Edo Period, the shrine was a thriving venue for entertainment such as sumo matches and theatrical performances, which were allowed in the shrine grounds. The theater was a type of Edo-sannomiya theater, and is said to have been first held inside Shiba-shinmei in 1645. In September of every year, the "Daradara Festival", an event supposedly named due to its long duration (daradara means lengthy), is held, during which ginger is widely sold inside the shrine and its vicinity, giving the shrine its nickname as the "Ginger Market". As famous as the "Betterazuke market" of Kodenma-cho, the festival was such an enjoyable event for Edo commoners that it was even depicted in ukiyo-e art. During the festival, ginger, chigibako, and amazake are sold. Chigibakos are oval boxes made of cypress, said to bring additional clothing when put into a cabinet, a good luck charm of sorts, whose name comes from the idea of a thousand pieces of wood (chigi) leading to a thousand pieces of clothing (chigi). Beneath the stairway is a savings mound built to commemorate Makino Motojirou's achievements in establishing the Real-estate and Savings Bank (Fudo-chokin-ginkou).

In July of the third year of the Ansei era, Dutchman Henry Heusken arrived in Shimoda together with Townsend Harris, acting as his right-hand man. He was active in the negotiation and signing of the Treaty of Amity and Commerce between the United States and Japan. His grave is a ward-designated historic landmark. Born in Amsterdam, Heusken traveled to America and, as Harris valued his ability in languages such as German and French, was then appointed in Japan. With a cheerful and openhearted disposition, well versed and discerning on the conditions in Japan, he was well known among the foreign residents of Edo. However, on his way back from the Akabane reception hall where he was interpreting for the Prussian ambassador and the Japanese delegation, he was attacked by a group of assassins near Nakano Bridge. He died at the age of 28 on January 15, 1861. Although the most famous old cherry tree of the old capital no longer exists, in spring the cherries and roses of Sharon on the temple grounds display magnificent blossoms.

Tozenji Temple was used as a diplomatic residence by the first British diplomatic envoy to Japan, Sir John Rutherford Alcock, and his successor, Sir Harry Smith Parkes. Although it was attacked by the Joi sect several times, the inner study and the entrance remain as they were at that time.