Nowadays smartphone is becoming multi-purpose device because it has more processing power at affordable cost. The trend of using smartphone for business, banking and everyday tasks has attracted research community to address security issues in smartphone applications and their communication with external systems. Due to their wide acceptability in community public, it is becoming trivial to use smartphone as an authenticating device for banking applications and access control management systems. Current legacy solutions used for Physical Access Control System (PACS) are combination of software and hardware to control the access of users to physical resources (rooms, offices, buildings etc). Most of them are using biometric or smart card as an identity token. The associated cost and limited freedom to customize these solutions to organizational needs open research areas for smartphone researchers to use them in PACS. In our research, architecture for PACS along with security protocol for smartphone is designed that is used for identity verification, authentication and authorization in PACS. The designed authentication protocol is an extension of two-factor authentication protocol.