One general principle of object representation in the brain is a hierarchy of cortical regions devoted to processing different object properties, from simple to complex, from local to the global. Along the pathway from posterior occipital to inferior temporal cortex, neurons evolve from tuning to local features such as lines and edges to selective for objects like faces and words. This dissertation will present three empirical studies that investigated the hierarchical representation of faces and words along the ventral pathway. The first study focused on the neural responses to faces from ones' own race than another race in both the occipital face area (OFA) and the fusiform face area (FFA). The second study addressed the neural bases of invariant face identity representation, and showed that the representation of facial identity might involve face-sensitive areas beyond the core face network (i.e., the OFA and FFA), in particular a face area in the anterior inferior-temporal cortex. The third study demonstrated the differential functions of two word-related areas, a lateral occipital area and the so-called Visual Word Form Area (VWFA), in encoding visual structures of words. Together, the cumulative evidence suggests the existence of a series of cortical regions involved in processing of these two categories of objects, faces and words, for which people have extensive expertise. The functional organizations for these two object-pathways share a significant amount of common features, presumably reflecting similar computational principles required.